1
CAPE: Clinical Analytics Prediction Engine
Session #310, February 15, 2019
Chad Konchak, MBA, Assistant Vice President, Clinical Analytics
Nirav Shah, MD, MPH; Infectious Diseases
NorthShore University HealthSystem
2
Chad Konchak and Nirav Shah
Has no real or apparent conflicts of interest to report.
Conflict of Interest
3
• Current State of Predictive Modeling
• Vision of CAPE
• Modeling Performance
• EMR Integration and Prospective Modeling
• From predictive to prescriptive modeling and the learning health
system
• Managing Change and Lessons Learned
Agenda
4
Learning Objective 1
• Define, clearly, the problem with healthcare’s current state of predictive modeling implementations and
how they often fail to support clinical workflows and describe the CAPE framework for how to bring
multiple predictive models into a single prescriptive engine
Learning Objective 2
• Describe an inventory of key patient outcomes to predict and how to achieve a high accuracy for
prediction including both retrospective and prospective validation processes
Learning Objective 3
• Demonstrate the importance of tightly integrated predictive models into the EHR using real-time
processing via the Predictive Model Markup Language (PMML) including implications for displaying the
results and risk factors of a model to front-line clinicians
Learning Objective 4.
• Discuss the implications of a learning health system and how CAPE can help to achieve a better
understanding of the impactability of patient populations based on multiple risk models and propose
specific intervention bundles catered to the needs of that population
Learning Objective 5.
• Discuss the key cultural implications that an integrated predictive engine is able to facilitate and how it
can enable the care team to improve patient outcomes while lowering costs
Learning Objectives
5
• 4 Hospitals
• 950 Beds
• 9000+ Employees
• 2700 Physician Medical Staff
• 900+ Employed Physician Medical Group
• 60,000 Annual Admissions
• 1.8 Million Annual Office Visits
• 125,000 Annual ED Visits
• $100M+ Research Institute
• HIMSS stage 7 Inpatient & Ambulatory
• H&HN Most Wired 15 years in a row
6
Data enabled population healthcare delivery across the
care continuum.
Vision
Paradigm
11
22
Evolving from single siloed predictive models to a unifying risk profile
CAPE: Clinical Analytics Prediction Engine
Population level enhanced and targeted interventions
33
Collaboratively designed, prioritized and coordinated care through Epic
Shift
Phase I (Live September 2018): E-Cart*, Mortality and Readmission
WIP: Medical and Surgical Complications and Prospective Utilization
Timeline
*A predictive model designed in partnership with University of Chicago based upon NorthShore patient
population to detect patient deterioration. All other models described were developed by NSUHS
7
Our Patients
Low Mortality Risk
High Mortality Risk
Mr. Smith is a 80 year old man
with metastatic bile duct cancer
with failure to thrive and
progressive disease.
Pt is admitted for pneumonia
8
Predictive Modeling – Risk Stratification
Without
predictive
modeling
With predictive modeling
Homogenous
interventions
Targeted
interventions
9
The Potential of CAPE for Mr. Smith
Transfer to the ICU with
respiratory failure and
sepsis. Mr. Smith
requires intubation and
prolonged ICU stay.
eCART warning leads to
early intervention allowing
patient to stay on the floor.
In-hospital mortality
identifies patient, prompting
revisiting advanced care
planning and revising goals
of care. A prolonged ICU
stay is avoided.
Without
CAPE
The Potential of
CAPE
10
High eCART Risk!
High Readmission Risk!
How is the Paradigm Shifting?
1. What needs to be done?
2. Who needs to do the intervention?
3. How fast does the intervention need to be performed?
High Mortality Risk!
11
Deploying Interventions
CAPE will take all risk scores into account, identify patients with a care gap, present checklists of
interventions to the right caregivers and assign a priority of how quickly the task needs to be
performed all within EPIC
Ecart
Mortality
Readmission
CAPE
11
Individual predictive
models
22
CAPE Integration
33
Interventions Identified
44
Interventions triaged
Across care team
Hospitalist
Floor Nurse
High Priority Patient!
In Next Hour
ICU Evaluation
Q2h Vitals
Within 24 hours
ACP & Goals of Care
Before Discharge
Med to Bed
After Discharge
PCP Appointment outreach
Chaplain
PCP
Pharmacist
12
Building a Predictive Model – In The EMR
Gather Data
Variable Selection
Data
NorthShore EDW
Model developed and
exported to PMML
+ Points for Age
- Points for Female
+ Points for each Comorbidity
+ Points for Prior admissions
+ Points for high Sodium
Etc….
EMR Import
13
Train, test, and evaluate performance
Sensitivity:
If we Flag the top 5% we capture 45% of ALL mortality
Ranking patients arbitrarily without this model we would only catch 5%
We also can say, then, this model is 9 X better than “guessing”
PPV
The prevalence of the outcome in the top 5% is 20%
The prevalence in the entire population is 2%
We also can say, then, these patients are 10 times more likely to die
AUC: 0.9
The AUC takes into account how often
your model “got it right”:
An AUC of 0.5 = flipping a coin
You want to be above at least .65 (but
as usual, it depends)
An AUC of 0.9 is really, really good!
14
Model performance
1
– Phase I
The information contained in this document is privileged and confidential under The Medical Studies Act (MSA), 735 ILCS 5/8-2101, et. seq.
Predictive model AUC
PPV
2
Sensitivity
Lift
eCART cardiac
arrest
3
0.75
0.08
0.45 7.5
In
-hospital mortality 0.89
0.13
0.66 6.6
30
-day out-of-hospital
mortality
0.85
0.16
0.45 4.5
90
-day out-of-hospital
mortality
0.85
0.26
0.42 4.2
180
-day out-of-hospital
mortality
0.85
0.36
0.38 3.8
30
-day readmission 0.72
0.30
0.26 2.6
1
Model performance is likely to change after final Epic build adjustments
2
ePPV, sensitivity and Lift are measured at the 10
th
%-le of the population
3
eCART performance is based on Feb. 2017 testing data
15
Prospective Validation
• Typical model validation stops after retrospective validation
• For CAPE, we had a “soft go-live” and monitored the models in
live production
• Evaluated model performance at two time periods
– 4 Hours upon ED Arrival
– 24 Hours after floor
• Operational decisions and model sign-off based on 4hr ED Model
16
Comparative model AUC
Retrospective vs Prospective Validation
0.93
0.92
0.78
0.83
0.76
0.72
0.71
0.69
0.71
0.69
EDW 5/2016
-
9/2016
EDW 5/2017
-
9/2017
EDW 5/2018
-
9/2018
EPIC,
FLOOR+24
5/2018
-
9/2018
EPIC
ADM+4HR
5/2018
-
9/2018
AUC
In-hospital AUC
"Readmission
AUC"
In-hospital and readmission model performance
17
Integrate Each Model into Patient Risk Profiles
• In the “Old Days” we would be done
• With CAPE, we need to do this for ALL our outcomes
1. E-CART Risk Score
2. In Hospital Mortality Risk Score
3. Post Hospital Mortality Risk Score
4. Readmission Risk Score
Now, every patient has a different risk score for 4 outcomes
Patient 1
In-Hospital Mortality
Post-Hospital Mortality
Readmission
In-Hospital Mortality
Post-Hospital Mortality
Readmission
Patient 2
Patient 3
In-Hospital Mortality
Post-Hospital Mortality
Readmission
And there are a LOT of different-looking types of patients
E-CART
E-CART
E-CART
18
Prescriptive Interventions Based on Risk
Risk
Identified
Priority
Care
Provider
Intervention
E-cart Red
<2 hours
Physician Assess, Code Status, ICU
E-cart Yellow
<30 mins
Nurse
q2hVitals x 8h, lactic acid,
accompanied off unit
Mortality TBD
SW/Chaplain
Identify ACP, PCP agrees w/
GOC? GOC and document
Risk
Identified
Priority
Care
Provider
Intervention
Readmission PTD
Pharmacist/
Primary MD
Med to Bed
Readmission
Within
48h post
discharge
CT office
Patient touchpoint to ensure
appropriate post discharge care
Readmission PTD CM High risk CM enrollment
19
Why are we doing this?
Prediction
modeling
Risk Stratification/
Segmentation
Common
Lexicon
Prescriptive
Interventions
Checklists
Standardizing
Care
Advanced
Analytics
Rapid
Learning
Learning Health
System
20
How do we know if we are making a
difference?
Knowledge captured
as byproduct of care
delivery experience
Methodological rigor
Effect size
21
Analysis design:
Normal patient care
Randomization
Double blind
No withholding of care
All built into electronic
medical record
Requires IRB approval
Intervention
Control
High risk of
outcome of
interest
Advanced Analytics – Pragmatic Study Design
22
Learning Health System
The development of a continuously learning
health system in which science, informatics,
incentives, and culture are aligned for
continuous improvement and innovation, with
best practices seamlessly embedded in the
delivery process and new knowledge captured
as an integral by-product of the delivery
experience
23
Learning Health System
Rubin, AMIA 2016
24
Early Results - eCart
• Cardiac Arrest Model (eCart)
• Scores >95
th
percentile were flagged red or highest risk.
Scores between 85-95
th
percentile were flagged yellow, or
intermediate risk
• ICU transfer was strongly urged for new red scores, but the
discretion of the treating physician could overrule. Yellow
score patients had increased frequency of vital signs on
the floor.
• Outcomes:
• Red score patients transferred to the ICU had a lower mortality when
compared with controls (18.4 vs 32.5%; Χ
2
p=0.0004)
• Time to ICU transfer decreased from 6.5 (IQR 21.8) to 2.2 (IQR 4.6)
hrs p=0.0001
25
Analysis design:
6 week randomization
30 high risk patients daily with FTE
to perform 20 interventions max
per day
900 Patients, 600 Intervention, 300
Control
Primary outcome: effect on all
cause readmission
Intervention
Control
High risk
readmission
Advanced Analytics – Post Discharge Phone Call
26
Compliance Rate
27
Compliance Rate
28
Lessons Learned – Compliance
• Understand key process metrics
• Resistance to change and standardization MD >> RN
• Thoughtful about workflows and user interface
• Steady and continuous messaging and education
• Data driven process employing mixed methods
• Quick and direct feedback loop
29
• Executive Support & Alignment with goals across health system
• Clear vision & ability to articulate this vision – CAPE Tour
• Data and Tech heavy project – Need to invest in
analytics/informatics
• Clear governance structure
• Persistence
• Need early quick wins
• Celebrate success
Culture and Change Management
30
• Innovation isn’t impossible in large organizations – but you’ll
need determination
• Being creative is great, but innovators turn creativity into output
• De–risk your idea as far as you can
• Learn collaboration – it’s not the same as teamwork
• Get out of the office
• Don’t expect everyone to say yes straightaway
